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Effectiveness  of  Computer-Graded  vs.  Instructor-Graded  Homework  Assignments 
in  an  Elementary  Spanish  Course:  A  comparative  study  at  two  undergraduate 
institutions 


ABSTRACT 

Researchers  at  the  United  States  Air  Force  Academy  and  Metropolitan  State  College  of 
Denver  collaborated  on  a  study  to  determine  the  effectiveness  of  computer-graded  vs. 
instructor-graded  homework  assignments  in  elementary  Spanish  courses  at  their 
respective  institutions.  Subjects  completed  one  or  the  other  type  of  online  activities  tied 
directly  to  the  textbook  of  the  course  and  accessed  via  the  online  ancillary 
MySpanishLab.  Subjects  were  randomly  assigned  to  activity  type  and  then  completed 
this  type  of  activity  for  the  duration  of  a  complete  chapter  of  the  text.  Both  subject 
groups  were  then  tested  by  administration  of  the  same  chapter  test  for  the  text  used  at 
their  institution.  Following  completion  of  the  chapter,  the  online  activities,  and  the 
administration  of  the  evaluation  instrument,  subjects  were  asked  to  complete  a 
satisfaction  survey  indicating  their  reactions  to  and  preferences  for  a  particular  type  of 
online  activity.  No  significant  differences  were  found  between  groups  on  chapter  test 
scores.  Nevertheless,  definite  differences  were  noted  indicating  strong  preferences  for 
one  type  of  activity  over  another.  In  addition,  some  striking  differences  between  subject 
responses  from  each  institution  provide  useful  insight  into  potential  adjustments  to 
curricular  planning  and  future  research  in  this  area. 
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Effectiveness  of  Computer-Graded  vs.  Instructor-Graded  Homework  Assignments 
in  an  Elementary  Spanish  Course:  A  comparative  study  at  two  undergraduate 
institutions 


Richard  Dabrowski,  Jean  W.  LeLoup,  &  Lunden  MacDonald 
Introduction 

The  demands  on  the  time  and  energy  of  the  foreign  language  instructor  seem  to 
be  increasing  while  the  resources  to  meet  those  demands  are  ever 
shrinking,  instructional  technology  may  be  seen  as  a  coping  mechanism;  however,  its 
increased  use  may  generate  unease  in  caring  educators  who  want  the  best  for  their 
students  and  who  feel  that  personalized  interaction  improves  learning  outcomes.  A 
general  question  arises  when  considering  technology  use  in  the  foreign  language  (FL) 
classroom:  does  technology  enhance  instructional  delivery  or  does  it  detract  from  the 
positive  role  of  the  instructor  with  regard  to  learning  outcomes? 

Part  of  the  answer  to  this  question  seems  to  rely  on  the  nature  of  the  instructional 
delivery  that  the  instructor  wants  to  give.  Traditional  logic  suggests  that  formative 
interaction  between  student  and  instructor  will  produce  the  best  results,  while 
summative  interaction  is  useful  only  in  the  domain  of  assessment.  Instructors  may  feel 
that  they  are  doing  a  “good  job"  only  if  they  provide  this  formative  interaction  to  the 
students,  and  they  rely  on  summative  data  to  prove  this  point. 

The  language-learning  technology  implemented  in  the  great  majority  of 
undergraduate  language  programs  today  challenges  this  outlook  a  bit.  Technological 
platforms  allow  instructors  to  choose  between  formative  and  summative  assignments- 
both  performed  within  the  platform-as  they  plan  their  instructional  strategy.  Typically, 
instructor-graded  assignments  are  considered  to  be  formative,  while  computer-graded 
assignments  are  viewed  as  more  summative  in  nature. 

There  are  advantages  and  disadvantages  to  both  types  of  activity.  Instructor- 
graded  assignments  offer  the  potential  for  more  nuanced  feedback  to  individual 
students,  and  they  allow  the  instructor  to  closely  monitor  student  progress.  However, 
this  type  of  assignment  is  much  more  time  consuming  for  instructors  to  evaluate  and 
diagnostically  edit,  and  there  is  often  a  delay  in  giving  the  feedback  to  students. 
Additionally,  students  must  return  to  the  activities  within  the  technological  platform  in 
order  to  make  corrections  and  complete  the  assignments,  thereby  completing  the 
formative  feedback  loop.  With  the  machine-graded  assignments,  the  feedback  is 
immediate  but  constrained  to  the  parameters  built  into  the  computer  program;  an 
answer  is  either  right  or  wrong,  according  to  the  pre-programmed  response.  In  addition. 
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the  instructor  must  enter  and  exit  multiple  avenues  of  the  technological  platform  in  order 
to  examine  the  individual  performance  of  any  given  student  for  a  particular  activity. 
Performance  indicators  are  purely  summative,  and  the  instructor  is  not  privy  to  the  path 
the  students  took  to  arrive  at  each  answer. 

Thus,  several  questions  emerge.  If  instructors  want  to  deliver  the  best  instruction, 
and  if  they  favor  formative  techniques  over  purely  summative  ones,  does  it  make  sense 
to  use  computer-graded  activities  over  instructor-graded  activities?  Does  the  type  of 
student  using  the  activities  determine  the  efficacy  of  the  formative  versus  summative 
approach?  Does  the  use  of  impersonal  technology  detract  from  the  positive  presence 
of  the  instructor,  or  does  it  actually  allow  the  instructor  to  create  more  time  for  in-class 
oral  interaction,  lesson  planning,  etc.? 

This  study  investigates  how  two  different  institutions  used  an  online  package  of 
tutorials  and  homework  assignments  called  MySpanishLab  to  support  their  elementary 
Spanish  programs  and  to  attempt  to  answer  the  preceding  questions.  MySpanishLab 
offers  both  instructor-graded  and  computer-graded  assignments.  The  overarching 
purpose  of  this  study,  then,  was  to  determine  whether  the  extra  effort  required  by  both 
instructors  and  students  to  complete  the  formative  instructor-graded  assignments 
produced  any  significant  difference  in  student  learning  when  compared  to  students  who 
exclusively  performed  computer-graded  assignments. 

The  Institutions 

The  two  universities  participating  in  the  study  represent  vastly  different 
organizations  for  teaching  and  learning.  While  technology  is  indeed  a  tool  for 
diversification,  the  emphasis  of  the  current  study  was  on  how  students  would  react  to  a 
fixed  point  of  technology,  that  being  the  instructor-graded  versus  computer-graded 
activities.  In  essence,  the  study  came  to  be  about  how  diverse  student  bodies  reacted 
to  one  singular  aspect  of  pedagogical  technology.  The  diversity  of  the  two  institutions 
plays  a  key  role  in  the  outcomes  of  the  experiment. 

A  comparison  of  the  two  institutions  participating  in  this  project  is,  in  itself,  a 
study  of  contrasts.  The  United  States  Air  Force  Academy  (USAFA)  is  an  undergraduate 
institution  and  one  of  five  United  States  Service  academies.  The  application  and 
admission  processes  are  extremely  competitive  and  the  acceptance  criteria  are  highly 
selective;  acceptance  rates  are  generally  between  11-15%.  The  approximately  4400 
cadets  come  from  all  over  the  U.S.  and  some  foreign  countries,  including  several 
exchange  cadets.  The  vast  majority  of  cadets  fall  in  the  age  category  of  "traditional 
college  students,”  being  from  18-23  years  of  age  with  a  few  exceptions  due  to 
enrollment  of  prior-enlisted  personnel  and  transfer  students.  Cadets  are  required  to  live 
on  campus,  they  have  additional  military  and  athletic  responsibilities  and  various  further 
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restrictions  to  college  life  apply,  depending  on  academic  standing  and  year  of  the 
students.  Nearly  all  cadets  graduate  in  the  stipulated  four  years;  the  very  few 
exceptions  to  this  are  due  primarily  to  extremely  extenuating  circumstances.  All  cadets 
graduate  with  a  Bachelor  of  Science  degree. 

While  no  FL  major  is  possible  at  the  institution,  cadets  may  choose  to  minor  in  a 
FL;  the  minor  is  comprised  of  at  least  four  courses  at  the  200-level  or  above  with  a 
specified  grade  point  average.  USAFA  offers  coursework  and  the  minor  in  eight 
different  languages:  Arabic,  Chinese,  French,  German,  Japanese,  Portuguese, 
Russian,  and  Spanish.  USAFA  has  a  comprehensive  FL  requirement  that  is  applied 
according  to  academic  major.  Those  cadets  with  a  technological  major  (e.g.,  all  types 
of  Engineering,  Physics,  and  Aeronautical  Engineering)  must  complete  two  semesters 
of  a  foreign  language,  while  those  with  a  major  in  non-technical  areas  (e.g., 
Management,  Political  Science,  Foreign  Area  Studies)  must  complete  a  four-semester 
sequence.  All  incoming  cadets  take  a  FL  placement  exam  and  are  placed  in  the 
appropriate  FL  class  to  begin  their  required  sequence.  Those  with  sufficient  proficiency 
can  validate  the  FL  requirement  and  may  opt  out  of  further  FL  study. 

In  terms  of  instructor  personnel,  the  Department  of  Foreign  Languages, 
combined  with  the  Department  of  International  Programs  (whose  personnel  teach  at 
feast  one  FL  course  per  semester)  at  USAFA  is  one  of  the  largest  in  the  institution  with 
more  than  60  full-time  instructors.  Civilian  and  military  personnel  (at  various  academic 
and  military  ranks)  each  comprise  approximately  half  of  the  instructional  pool.  The 
general  faculty  split  at  USAFA  as  a  whole  is  approximately  70%  military  and  30% 
civilian  instructors.  While  the  curriculum  at  USAFA  has  a  strong  research-oriented 
component  and  instructors  are  expected  to  facilitate  same,  USAFA  is  self-designated  a 
“teaching  institution.” 

While  more  traditional  in  terms  of  student  body  and  academic  offerings, 
Metropolitan  State  College  of  Denver  (Metro  State)  is  a  unique  institution  in  Colorado 
and  the  United  States.  The  urban  college  shares  the  downtown-Denver  Auraria 
Campus  with  two  other  institutions  of  higher  education  (the  Community  College  of 
Denver  and  the  University  of  Colorado  at  Denver),  and  common-space  agreements 
among  the  three  allow  Metro  State  students  to  take  advantage  of  both  preparatory  and 
more  advanced  offerings  at  the  respective  neighbor  schools.  Three  Master’s  Degree 
programs  have  recently  been  added  to  Metro  State’s  wide  slate  of  baccalaureate 
degree  offerings. 

Metro  State’s  student  body  of  over  24,000  is  an  exemplar  of  diversity.  The 
institution  admits  students  via  a  modified  open-enrollment  policy  (completion  of  the 
General  Equivalency  Degree  or  traditional  high  school  curriculum  program  and 
minimum  age  are  the  baseline  requirements),  and  the  student  age  range  is  from  16  to 
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70,  with  the  median  age  of  students  being  23  years.  Metro  State  students  are  typically 
high  academic  achievers  who  struggled  in  the  traditional  academic  setting  and  came  to 
Metro  State  in  search  of  a  more  flexible,  individualized  approach  to  higher 
education.  Metro  State  also  enrolls  many  non-traditional  students  who  are  returning  to 
school  after  military  service,  career,  family,  incarceration,  or  other  life  experiences 
outside  of  the  academy.  With  limited  residential  assistance  for  students,  Metro  State  is 
considered  a  commuter  college,  and  almost  all  students  work  part-  or  full-time  in 
addition  to  their  studies.  This  notwithstanding,  Metro  State’s  reputation  as  a  leader  in 
educational  quality  is  drawing  many  traditional  students  to  the  campus  straight  from 
high  school.  Regardless  of  their  origin,  94%  of  the  student  body  hails  from  the  seven- 
county  Denver  metropolitan  area,  and  Metro  State  is  clearly  a  popular  “school  of  choice” 
for  Colorado  students. 

With  the  exception  of  select  programs  (Journalism  and  Hospitality),  Metro 
State  does  not  have  a  language  requirement  for  the  baccalaureate  degree.  Regardless 
of  this  fact,  the  Department  of  Modern  Languages  (MDL)  at  Metro  State  offers  a 
traditional  undergraduate  program  in  language  and  literature  to  a  robust 
enrollment.  Majors  are  offered  in  Spanish,  French  and  German,  with  minor  programs 
and  introductory  courses  in  other  languages.  The  MDL  offers  several  first-  and  second- 
year  courses  online,  and  a  Translation  Certificate  program  at  the  upper-division 
level.  In  conjunction  with  the  Metro  State  Teacher  Education  Program,  the  department 
offers  a  unique  major  designed  to  support  students  who  wish  to  use  FL  as  a  primary  or 
ancillary  support  in  their  licensure.  Courses  specifically  designed  for  heritage  speakers 
are  also  offered.  In  addition  to  on-campus  study,  MDL  develops  and  offers  multiple 
study  abroad  programs  that  are  highly  attended.  Language  learning  technology  is  used 
in  all  first-  and  second-year  courses,  and  the  majority  of  upper-division  courses  also 
implement  some  type  of  language  learning  technology. 

Metro  State  is  currently  seeking  status  as  a  Hispanic  Serving  Institution 
(HSI).  The  current  enrollment  of  17.6%  Hispanic  students  in  the  year  201 1  earned 
Metro  State  national  recognition  as  an  “emerging  HSI,”  and  the  institution  is  consistently 
ranked  among  the  top  100  U.S.  institutions  for  graduating  Latino  students  (College  Fact 
Sheet,  n.d.).  The  goal  of  achieving  HSI  status  plays  a  huge  role  in  the  development  of 
curriculum  and  programs  at  the  College,  and  this  is  particularly  evident  in  the  Spanish 
Program  within  MDL.  Taking  into  consideration  traditional  majors,  minors,  double¬ 
majors  and  students  with  an  emphasis  in  teaching,  almost  four  percent  (3.9%)  of  the 
total  student  body  is  currently  enrolled  in  Spanish  classes  at  Metro  State. 

The  FL  instructor  base  at  Metro  State  is  comprised  of  nine  full-time  tenured 
and/or  tenure-track  faculty  members  and  between  20  and  30  active  “Affiliates” 
(contingent  faculty).  Approximately  56%  of  full-time  faculty  teaches  introductory  and 
lower- intermediate  classes  on  a  regular  basis.  Of  the  Affiliate  group,  up  to  80% 
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regularly  teach  at  other  institutions.  Though  full-time  faculty  members  are  expected  to 
fulfill  the  typical  university  requirements  for  research  and  service,  Metro  State  is,  like 
USAFA,  considered  a  “teaching  institution."  Within  MDL,  however,  research  is  most 
often  performed  in  the  area  of  the  Scholarship  of  Teaching  and  Learning  in  order  to 
maximize  research  talent  while  adhering  to  the  pedagogical  goals  of  the  department 
and  the  teaching  mission  of  the  College. 

The  Study 

Rationale 

While  language-learning  technology  is  de  rigueur  in  the  majority  of  FL  programs 
today,  when  the  current  study  began  in  2010  the  adoption  of  a  program  as  extensive  as 
MySpanishLab  was  exceptional.  Both  USAFA  and  Metro  State  considered  the  program 
to  be  cutting-edge,  but  both  schools  had  many  questions  regarding  the  efficacy  of  a  tool 
that  had  yet  to  be  fully  considered  or  proven  by  the  educational  market.  Additionally, 
both  schools  were  curious  about  how  this  type  of  technology  would  influence  the 
evolution  of  traditional  teaching  methodologies  in  undergraduate  FL  classes. 

Specifically,  USAFA  was  interested  in  performing  this  study  with  the  intention  of 
justifying  the  decision  to  select  the  jAnda!  textbook  series  (Heining-Boynton  &  Cowell, 
2009;  Heining-Boynton,  LeLoup,  &  Cowell,  2010)  that  was  accompanied  by  the 
MySpanishLab  suite  of  online  reinforcement  activities.  The  impetus  for  Metro  State 
joining  the  study  was  similar — the  j Arriba!  textbook  series  (Zayas-Bazan,  Bacon,  & 
Nibert,  2012)  had  already  been  adopted  by  MDL,  but  researchers  wanted  to  justify  the 
decision  to  incorporate  MySpanishLab  into  a  program  previously  devoid  of  language 
learning  technology. 

Both  universities  had  turned  to  the  literature  to  inform  their  adoption  and 
implementation  decisions  prior  to  undertaking  the  current  study,  but  the  results  of  this 
research  had  provided  theoretically  contradictory  answers  as  to  the  potential  efficacy  of 
language  learning  technology.  Blake  (2008)  takes  the  view  that  technology  can 
enhance  second  language  acquisition  depending  on  how  it  is  used  in  the  curriculum, 
which  can  include  accessing  the  Internet  as  a  source  of  authentic  content,  social 
interaction  via  computer-mediated  communication  in  the  target  language,  and  learner 
feedback  through  the  use  of  computer-assisted  language  learning  (CALL)  activities. 
While  all  these  uses  are  potentially  beneficial,  the  jAnda!  model  of  combining  classroom 
instruction  with  tutorial  CALL  was  particularly  appealing  due  to  its  apparent  efficient 
management  of  resources,  specifically  instructor  and  learner  time  and  effort. 

MacDonald  (2011)  describes  how  commercially-produced,  textbook  publisher-supported 
virtual  language  laboratories  seem  to  help  compensate  for  institutional  and  student 
challenges  such  as  providing  more  consistent  quality  among  course  sections  and 
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offering  additional  assistance  to  those  students  who  need  it.  However,  Ushida  (2005) 
found  that  students  may  have  anxiety  about  online  language  learning  that  could  affect 
their  performance.  Also,  instructors  in  different  sections  of  the  same  course  can 
implement  the  online  components  idiosyncratically,  which  may  also  have  an  impact  on 
student  learning.  Bush  (2008)  amplifies  the  idea  that  while  the  benefits  of  computer- 
assisted  language  learning  may  seem  to  some  to  be  obvious,  the  reality  is  that  some 
students  and  even  some  instructors  may  resist  making  effective  use  of  this  resource. 

For  the  current  researchers,  the  aspect  of  MySpanishLab  that  was  potentially  the 
most  troubling  was  that  it  gives  the  course  director  the  option  of  assigning  computer- 
graded  activities,  instructor-graded  activities,  or  a  mixture  of  the  two.  Assigning  only 
computer-graded  activities  is  appealing  as  it  frees  instructors  from  the  tedious  and  time- 
consuming  task  of  providing  homework  feedback  to  students,  though  possibly  at  the 
expense  of  short-changing  students  of  the  potentially  detailed,  nuanced  and 
individualized  error  corrections  that  a  caring  instructor  is  able  to  provide.  Research 
(Nagata,1996;  Hubbard  &  Siskin,  2004;  Goodwin-Jones,  2009)  indicates  that  well- 
designed  computer  feedback  can  be  more  effective  than  manually-graded  homework 
assignments  in  producing  significant  differences  in  learning,  especially  in  helping 
students  practice  and  learn  grammatical  constructions.  But  not  all  instructors  and 
students  may  be  convinced  of  this — hence  the  value  of  this  study. 

The  ready  availability  of  two  treatments  [computer-graded  vs.  instructor-graded] 
lent  itself  to  designing  a  comparative  research  study  where  each  treatment  could  be 
assigned  to  a  separate  group  and  the  effects  measured  using  a  single  assessment  for 
the  entire  population.  The  most  similar  published  study  was  the  dissertation  by 
Echavez-Solano  (2003),  which  compared  measures  of  beginning  Spanish  student 
motivation,  aptitude  and  proficiency  over  a  semester  between  an  experimental  group 
with  a  hybrid  of  web-based  and  classroom  activities  and  a  control  group  that  only  had 
classroom  activities.  Her  findings  were  that  no  significant  differences  existed  in  either 
student  performance  or  higher  satisfaction  among  the  experimental  group  students. 
Scida  &  Saury  (2006)  also  looked  at  student  performance  in  hybrid  elementary  Spanish 
courses  with  results  consistent  to  those  of  Echavez-Solano.  Similar  comparative 
research  done  with  other  languages  has  shown  similar  findings  (Allum,  2002; 
Chenoweth  &  Murday,  2003;  Peters,  Weinberg,  &  Sarma,  2008). 

Replicating  aspects  of  this  earlier  work  seemed  to  be  the  most  reliable  means  of 
assessing  the  value  of  the  computer-graded  versus  instructor-graded  activities  in 
MySpanishLab  at  both  USAFA  and  Metro  State.  For  the  current  study,  research 
questions  and  objectives  were  collaboratively  developed,  and  the  study  was  conducted 
at  both  institutions. 
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Research  questions 


Both  institutions  shared  the  overall  research  question:  Is  there  a  significant 
difference  in  language  proficiency/performance-as  demonstrated  on  a  FL  test— between 
students  learning  with  computer-graded  assignments  versus  those  learning  with 
instructor-graded  assignments?  Additional  components  of  the  study  queried  preference 
for  assignment  type  by  both  students  and  instructors.  Researchers  hypothesized  that 
there  would  be  no  significant  difference  in  performance  between  the  two  groups. 
Additionally,  at  Metro  State,  it  was  hypothesized  that  subjects  (Ss)  and  the  pluri- 
employed  instructor  staff  would  prefer  the  computer-graded  activities. 

Research  Design  and  Assessment  Strategy 

In  the  USAFA  study  the  Ss  were  229  students  of  elementary  Spanish  divided  into 
14  sections.  Subjects  in  seven  sections  (Ss=1 15)  were  assigned  to  complete  only  the 
machine-graded  assignments  of  MySpanishLab,  while  Ss  (11 1)  in  the  other  seven 
completed  only  the  instructor-graded  assignments.  The  number  of  computer-graded 
activities  was  approximately  three  times  the  number  of  the  instructor-graded  activities, 
because  the  computer-graded  activities  are  closed-ended  (True/False,  multiple  choice, 
etc,),  which  take  less  time  to  complete  than  the  open-ended,  instructor-graded  activities. 
Example  questions  for  each  type  of  activity  are  included  in  Appendix  A.  Based  on 
archival  data  showing  time  on  task  from  the  MySpanishLab  database,  this  division  of 
the  number  of  activities  was  deemed  to  equalize  roughly  the  time  it  would  take  each 
group  of  Ss  to  complete  their  particular  activities. 

Where  the  same  instructor  taught  more  than  one  section,  that  instructor  had  at 
least  one  section  completing  each  type  of  activity.  The  three-week  period  of  the  study 
corresponded  to  the  lesson  plans  for  Chapter  5  of  the  jAnda!  textbook  used  in  the 
elementary  Spanish  course.  Upon  completion  of  chapter  instruction  and  activities,  all 
students  took  the  same  chapter  exam.  The  results  were  analyzed  by  sections  to 
determine  any  significant  difference  between  the  two  groups  in  scores  on  the  chapter 
test.  During  the  lesson  following  the  exam,  students  were  asked  to  fill  out  an  online 
satisfaction  feedback  form  (Appendix  B)  to  determine  how  well  they  liked  the  approach 
taken  by  their  section  toward  online  assignments  for  this  chapter. 

Parameters  for  the  Metro  State  study  were  slightly  different.  Researchers  at 
Metro  State  were  unable  to  mandate  participation  in  this  study,  and  only  a  small 
percentage  of  instructors  volunteered  to  participate  in  the  instructor-graded  side. 
Instructor  participants  came  from  two  levels  of  the  introductory  sequence:  1010  and 
1020.  The  instructors  who  chose  to  participate  did  so  by  performing  instructor  grading 
for  one  chapter  of  exercises.  The  number  of  computer-graded  activities  assigned  was 
approximately  seven  times  the  number  of  instructor-graded  activities.  However,  as  at 
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USAFA,  due  to  the  open-ended  nature  of  the  instructor-activities  assigned  it  was 
estimated  that  all  students  would  spend  roughly  the  same  amount  of  time  performing 
the  activities  (see  Appendix  A).  In  all,  126  out  of  41 1  students  (44%)  enrolled  in  both 
levels  completed  instructor-graded  activities. 

All  students  in  the  program  took  the  same  quiz  at  the  end  of  the  chapter  (from  the 
jArriba!  text,  Chapter  5  [1010]  and  11  [1020]),  and  numbers  were  compared 
accordingly.  The  Metro  State  analysis  followed  the  USAFA  model,  and  all  students  in 
all  sections  were  asked  to  fill  out  a  satisfaction  survey  about  MySpanishLab  (using  the 
same  form  as  USAFA). 


Results  and  discussion 

At  USAFA,  an  analysis  of  results  on  the  chapter  test  yielded  a  confirmation  of  the 
initial  hypothesis.  No  significant  difference  in  performance  existed  between  the  two 
groups.  In  fact,  the  average  chapter  test  scores  overall  were  nearly  identical  (see  Table 
1).  Metro  State  results  yielded  a  similar  confirmation  of  the  initial  hypothesis. 

Table  1 

Comparison  of  test  results  between  treatment  groups 


Treatment  group 

USAFA  test  scores 

Metro  test  scores 

Instructor-graded  group 

158.68/200  (SPA  132) 

62.44/  100  (SPA  1010) 

64.9/  100  (SPA  1020) 

Computer-graded  group 

158.48/200  (SPA  132) 

63.1  /  100  (SPA  1010) 

52.9/  100  (SPA  1020)* 

"Note  for  Table  1:  An  important  consideration  when  examining  these  results-  Metro  students  who  dropped  the  course  but  had  not 
been  inactivated  in  MySpanishLab  by  their  instructors  were  included  in  testing  statistics  as  a  0%  grade. 

A  comparison  of  the  survey  results  from  both  institutions  yields  the  most  telling 
difference  to  be  in  the  preference  for  type  of  online  assignments.  Following  testing, 
students  at  both  schools  were  asked  to  submit  answers  to  a  satisfaction  survey  (see 
Appendix  B.  The  USAFA  response  to  the  satisfaction  survey  was  at  65%  (N=146).  At 
Metro  State,  52.5%  of  Ss  responded  to  the  satisfaction  survey  (N  =  216).  Among  other 
questions,  the  Ss  were  asked  to  state  their  preferences  for  activity  type:  computer- 
graded,  instructor-graded,  or  a  hybrid  of  the  two.  The  results  from  the  USAFA  survey 
are  strikingly  different  from  those  obtained  by  the  Metro  State  study  (see  Table  2). 
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Table  2 

Comparison  of  survey  results  between  institutions 


USAFA  Survey  preferences 

Metro  State  Survey  preferences 

24%  preferred  hybrid 

68%  preferred  hybrid 

70.5%  preferred  computer-graded 

9.7%  preferred  computer-graded 

5%  preferred  instructor-graded 

18.5%  preferred  instructor-graded 

.5%  no  opinion 

1  3.8%  no  opinion 

interestingly,  the  Ss’  performance  on  the  chapter  test  did  not  reveal  any  significant 
difference  in  scores  at  either  institution.  Nevertheless,  student  learning  preferences 
should  not  be  discounted  or  summarily  dismissed  (Lightbown  &  Spada,  2006).  Subjects 
in  both  studies  indicated  an  overwhelming  preference  for  a  specific  type  of  online 
activity,  albeit  different  at  each  institution. 

The  explanation  for  the  stark  differences  in  survey  responses  may  lie  in  the 
nature  of  the  institutions  and  their  respective  student  bodies.  Cadets  at  USAFA  are 
under  severe  time  constraints  and  are  hard-pressed  to  complete  all  homework 
assignments  for  their  heavy  course  load  on  a  regular  basis.  Perhaps  the  expediency  of 
computer-graded  activities  meets  their  particular  needs  in  so  far  as  course  preparation. 
This  type  of  activity  is  clearly  less  time-consuming  to  complete  than  the  category  of 
instructor-graded  assignments,  which  necessitate  a  revisiting  of  each  task  to  review, 
repair,  and  resubmit.  Researchers  at  Metro  State  attribute  the  deviation  of  preference 
from  students  at  USAFA  to  student  body  composition  and  “personality”  of  their  learners. 
Students  at  Metro  State  are  very  concerned  with  their  learning  and  are  paying  directly 
for  their  education.  While  many  have  time  constraints  just  as  compelling  as  those  of 
USAFA  students,  perhaps  they  are  more  concerned  with  "getting  their  money’s  worth” 
out  of  their  education.  With  this  mindset,  they  appreciate  both  the  rapid  feedback 
provided  by  computer-graded  activities  as  well  as  the  individual  attention  and  instruction 
garnered  through  the  instructor-graded  tasks. 

One  rather  troubling  and  perhaps  telling  result  reflected  in  the  survey  was  the 
USAFA  response  to  question  #6,  directed  toward  those  Ss  engaging  with  the  instructor- 
graded  activities. 

6.  If  you  completed  the  Instructor-graded  activities  for  Chapter  5,  how  often  did 

you  read  the  instructor  feedback  provided  (please  circle): 


Every  Time  Most  of  the  Time  About  Half  the  Time  Rarely  Never 
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Well  over  half  of  the  Ss  responding  to  this  question  indicated  that  they  “rarely”  or  “never” 
read  the  instructor  feedback.  Clearly,  if  students  are  not  willing  to  seek  and  reap  the 
benefit  of  nuanced  personalized  instruction,  such  as  that  provided  by  individual 
instructor  comments  particular  to  student  production,  it  begs  the  question  of  the  worth  of 
such  additional  and  extensive  effort  on  the  part  of  the  instructor. 

Qualitative  data  extracted  from  the  satisfaction  survey  at  both  institutions 
provided  additional  information  about  Ss’  preferences  and  dislikes  vis-a-vis 
MySpanishLab.  Tables  3  and  4  address  survey  questions  8  and  9  in  particular. 

Table  3 

Survey  question  #8:  What  do  you  like  most  about  MySpanishLab? 


USAFA 

Metro  State 

•  30%  Learning 

•  28%  Feedback 

•  16%  Ease/Availability 

•  13%  Specific  Features 

•  Miscellaneous  responses 

•  22.2%  Instant  feedback 

•  10.1%  Extra  practice 

•  9.3%  Ease/availability 

•  8.3%  Work  at  your  own  pace 

•  6.0%  Multiple  attempts  at  same 
task 

•  Miscellaneous  responses 

Table  4 

Survey  question  #9:  What  do  you  like  least  about  MySpanishLab? 


USAFA 

Metro  State 

•  37%  Specific  Features 

•  34%  Quantity 

•  10%  Timing/Time  Spent 

•  8%  Learning 

•  7%  Quality 

•  26.9%  Time  consuming 

•  21 .3%  No  answer 

•  13%  Frustrating 

•  1 1 .1  %  Too  many  assigned  activities 

•  4.2%  No  comprehensive  feedback 

•  Miscellaneous  responses 

For  survey  question  #9:  responses  in  this  category  included  many  complaints  of 
a  technological  nature  (unforgiving  computers  as  opposed  to  understanding  human 
beings;  inconsistent  Internet  availability)  as  well  as  failure  of  completion  of  instructor- 
graded  activities  on  both  students’  and  instructors’  parts.  It  is  noteworthy  that  Metro 
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State  Ss'  second  highest  response  to  question  #9  was  “no  response";  in  other  words, 
these  Ss  had  no  dislikes! 

No  data  were  gathered  to  reflect  instructor  satisfaction,  but  some  deductions  can 
be  made  from  results  of  the  studies  and  attempts  at  further  replication  of  the  same.  At 
USAFA,  several  Ss  failed  to  complete  the  survey  because  some  instructors  simply  did 
not  administer  it.  In  addition,  several  Ss  commented  in  the  survey  that  they  either 
received  no  instructor  feedback  or  what  they  did  receive  was  not  generated  in  a  timely 
manner.  In  other  words,  complete  instructor  participation  was  lacking  in  the  USAFA 
study.  At  Metro  State,  the  small  number  of  instructor  volunteers  is  representative  of  a 
reluctance  of  the  greater  number  of  instructors  to  take  on  additional  grading.  It  would 
also  seem  to  support  the  initial  hypothesis  of  their  preference  for  computer-graded 
activities.  When  queried  as  to  willingness  to  participate  in  a  follow-up  study,  no  one 
volunteered.  This  refusal  to  continue  with  the  study  speaks  volumes  in  terms  of 
instructor  prioritization  of  time,  preference  for,  and/or  confidence  in  the  employment  of 
computer-graded  activities  for  positive  student  progress. 


Lessons  learned 

A  number  of  lessons  were  learned  as  a  result  of  this  joint  study.  First,  steps 
need  to  be  taken  to  ensure  the  full  cooperation  of  all  instructors  involved  in  any  future 
study  of  this  ilk.  They  must  “buy  into”  the  procedures  completely  to  facilitate  optimal 
performance  of  their  students  and  maximum  collection  of  relevant  data.  As  participation 
in  the  study  at  Metro  State  was  voluntary,  the  instructors’  desire  to  be  included  can  be 
considered  a  key  finding — initial  curiosity  during  the  first  semesters  of  the  adoption  of 
this  technology  was  high,  and  the  need  for  understanding  its  impact  seemed 
urgent.  Nevertheless,  their  refusal  to  participate  in  the  study  in  subsequent  semesters 
is  quite  telling  and  indicates  a  developed  preference  for  and  confidence  in  computer- 
graded  activities.  Next,  as  indicated  by  USAFA  student  questionnaire  responses,  finding 
a  way  to  require  students  to  access  and  read  instructor  feedback  must  be  developed.  If 
students  are  not  taking  advantage  of  the  additional  instructor  feedback  given  via  the 
instructor-graded  tasks,  it  is  difficult  to  tease  out  the  effect  or  lack  thereof  of  this  sort  of 
assignment  on  student  learning.  Third,  and  related  to  the  cooperation  of  all  instructors 
included  in  the  study,  an  improved  response  rate  on  the  student  satisfaction  survey 
would  yield  more  potentially  useful  data.  Finally,  restructuring  the  FL  curriculum  in  order 
to  optimize  use  of  instructor  time  while  concomitantly  generating  helpful  feedback  for 
students  would  seem  to  be  a  worthy  goal. 

While  the  variation  in  test  scores  between  students  who  performed  instructor- 
graded  versus  computer-graded  homework  exercises  is  roughly  similar  at  both  schools, 
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the  student  preferences  and  dislikes  regarding  MySpanishLab  as  a  learning  platform 
clearly  reflect  differences  in  the  respective  student  bodies. 

The  satisfaction  surveys  also  confirmed  some  anecdotal  opinions  that  had  been 
present  in  the  Spanish  section  of  MDL  at  Metro  State.  This  program  had  been  one  of 
the  first  in  the  nation  to  broadly  implement  the  MySpanishLab  platform,  and  many  initial 
"kinks”  in  the  system  were  being  solved  over  the  course  of  use  during  the  first  few 
semesters.  Students  expressed  frustration  with  these  technological  glitches  (e.g.  initial 
lack  of  interface  for  Mac  users,  the  need  [at  both  institutions]  to  make  sure  that  all 
applicable  plug-ins  and  other  in-system  tools  were  appropriately  installed  and  available 
in  student-use  computer  labs,  on  student-issued  computers,  etc.).  It  became  clear 
during  the  first  semesters  of  use  that  there  is  a  need  to  be  pro-active  and  vocal  when 
the  technology  does  not  work  and  a  need  for  dialog  with  the  publishers  to  identify  these 
errors  and  solve  system  malfunctions  in  future  iterations  of  the  platform.  Also  identified 
was  the  importance  of  reporting  what  does  work-in  other  words,  the  experience  of 
adopting  this  platform  and  the  student  response  to  its  use  has  highlighted  the 
importance  of  rejecting  “passive  technology  use.” 


Programmatic  Changes  Due  to  the  Study 

For  both  universities  the  study  highlighted  the  need  to  closely  examine  how 
language  learning  technology  is  used  in  FL  classrooms  in  the  future.  Despite  any 
questions  that  still  remained  following  the  examination  of  study  results,  both  universities 
were  able  to  identify  concrete  takeaways  that  directly  affected  program  design  and 
instructional  delivery. 

Given  the  lack  of  appreciation  for  instructor-provided  corrections  on  homework 
assignments  and  the  effectiveness  of  the  immediate  feedback  on  computer-graded 
exercises,  USAFA  discontinued  using  the  instructor-graded  MySpanishLab  activities  in 
the  elementary  Spanish  course.  The  course  was  reconceived  as  a  hybrid  of  classroom 
instruction  combined  with  computer-based  MySpanishLab  reinforcement  self-study. 
Approximately  30%  of  the  academic  calendar  was  set  aside  for  MySpanishLab  so  that 
cadets  could  do  the  activities  at  their  own  pace  during  class  time  under  the  supervision 
of  an  instructor.  If,  however,  cadets  were  absent  due  to  illness,  participation  in  sports 
or  any  other  reason,  they  could  do  the  work  wherever  they  could  find  Internet  access. 
Besides  freeing  instructors  from  the  drudgery  of  manually  grading  homework,  an 
additional  consideration  was  that  multiple  sections  meeting  during  the  same  class 
period  could  be  combined  in  a  small  lecture  hall  and  supervised  by  a  single  instructor, 
further  freeing  up  faculty  time.  Overall,  this  hybrid  arrangement  gives  students  more 
control  and  responsibility  for  their  own  learning  while  allowing  faculty  to  concentrate 
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their  time  and  energy  on  classroom  instruction  and  individual  tutoring  during  office 
hours. 


Similar  programmatic  changes  took  place  at  Metro  State.  As  student 
responses  indicated  a  clear  predilection  among  Metro  State  Students  for  instructor 
participation  in  their  work  in  combination  with  the  ease  and  instant  feedback  of 
computer-graded  activities,  subsequent  courses  at  Metro  State  were  designed  with  this 
hybrid  combination  of  workload  in  mind.  While  still  maintaining  the  uniformity  of  a 
consolidated  departmental  program,  individual  instructors  have  been  given  more 
flexibility  with  regard  to  augmenting  the  common  and  shared  base  of  computer-graded 
activities  with  instructor-chosen  and  instructor-graded  activities  that  they  individually 
assign.  Additionally  it  was  learned  that,  by  assigning  fewer  “busy  work"  activities,  the 
time  spent  on  computer-graded  activities  could  be  decreased  and  the  meaningfulness 
of  student  work  could  be  increased.  Consequently,  instead  of  formally  assigning  all 
activities  in  MySpanishLab  for  any  given  chapter  in  the  introductory  sequence  courses, 
the  program  coordinator  establishes  a  baseline  for  minimum  participation  that  does  not 
include  activities  identified  by  students  and  instructors  as  comparatively  less 
valuable.  The  study  results  allowed  Metro  State  to  make  an  informed  move  toward  the 
development  of  a  more  hybrid,  plural,  yet  individualized  approach  while  maintaining  the 
integration  of  technology  in  the  language  learning  classroom — in  essence,  the  study 
allowed  MDL  to  meet  the  needs  of  both  the  students  and  the  program. 

A  foundational  outcome  for  both  universities  was  that  the  way  any  program  is 
designed  around  technology  must  truly  fit  with  the  personality,  learning  styles,  and 
desires  of  student  learners.  In  order  to  do  this,  student  performance  and  preference 
must  be  continually  assessed,  and  programmatic  changes  must  be  implemented  in 
accordance  with  assessment  results. 

Conclusion 

It  is  interesting  to  speculate  on  the  reasons  for  the  data  differentials  that  did 
surface.  It  is  certainly  possible  that  these  differences  are  illustrative  of  the  dissimilarity 
of  the  two  student  populations.  Indeed,  defining  characteristics  of  the  Ss  such  as 
personality,  academic  preparation,  attitude  toward  the  institution,  and  even  reasons  for 
college  enrollment  may  figure  into  the  disparity  in  preference  for  online  activity  type. 
Some  stated  dislikes  due  to  technological  problems  can  obviously  be  ameliorated  by 
working  more  closely  with  the  publisher  of  MySpanishLab,  identifying  system 
malfunctions  and  errors  in  feedback.  The  resulting  data  do  inspire  further  inquiry  that 
may  lead  to  improved  teaching  and  learning  in  FL  classes  at  the  university  level. 

Questioning  the  efficacy  of  formative  versus  summative  activity  types  within 
language  learning  technology  platforms  can  illuminate  the  benefits  and  disadvantages 
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of  contemporary  textbooks  and  ancillary  materials  for  the  21st-century  learner.  More 
thoughtful  consideration  of  students’  activity  preference,  in  combination  with 
instructional  design  that  is  driven  by  positive  assessment  data,  can  enhance  the  already 
strong  teaching  being  performed  in  diverse  types  of  universities.  Responsible  and 
intelligent  use  of  language  learning  technology  can  offer  students  and  instructors  alike  a 
variety  of  choices  that  can  tailor  a  program  to  a  high  level  of  satisfaction  while  leveling 
the  field  in  terms  of  student  performance  and  learning  outcomes.  Further  research  along 
these  same  lines  has  the  potential  to  generate  more  fruitful  data  that  can  affect 
programmatic  change.  The  benefit  of  improved  FL  learning  for  students  will  be  a  most 
positive  result. 
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APPENDIX  A 


INSTRUCTOR-GRADED  ACTIVITY  SAMPLE 


SAH  OS  ""  'nvA?»ttoay  eseuchr 
TUTORIALS  i  E-BOOK 


5 LOSSARY  VERB  CHART 


SAV  E  ter  i*UT  f  I N  i  5 H :  Sub  fll  i  X  for  Gn 


a  e  l  6  0  u  n  i 


CAPS  LO 


A 


[Instructor  Graded] 

Use  the  Internet  to  find  Mana’s  official  web  site  in  order  to  listen  to  some  of 
their  music  and/or  view  some  of  their  videos.  To  get  an  idea  of  the  wide  array  of 
songs  that  they  have  composed,  you  could  also  go  to  an  online  store  that 
allows  you  to  hear  brief  clips  of  file  songs  on  their  CDs, 

Paso  1  After  hearing  at  least  some  of  three  differe  nt  songs,  fill  in  and  select  the 
following  information. 

1)  Cancidn  1 

Titulo:  _ _ „ 

Rttmo;  rapico  aiegre.  fianquilo,  suave,  lento 

Estiiot  rod;,  latino,  apasicnadc.  bailable 

Mi  parte  favorite  de  la  cancidn:  gu  naira  tatena  voz 

Ml  opinion  de  la  cancidn:  Me  guata  muctic.  Me  gusta  Es  buena.  No  es 
muy  buena.  No  me  gusta  Es  mala. 
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COMPUTER-GRADED  ACTIVITY  SAMPLE 


*U9$0ft  w  n  u  « 

my-:  pa  -  sr  lab 


tv.  ' 


SAM  05-14  iQue  >1  ico  compranias? 
TUTORIALS  I  E-BOOK  «  GLOSSARY 


VERB  CHART 


l_j  SAVE  lor  lam  FINISH; 

a  e  i  o  u  u  h  ^ 


Marta  and  Paco  have  a  gift  certificate  to  buy  a  new  CD,  but  Marla  would  like  to  buy 
one  CD  and  Paco  has  chosen  a  different  one.  Complete  their  dialogue  with  the 
correct  forms  of  este  and  ese 


n 

Marta:  PaC0'  mira-  el  dkG®  perfecte'  t  1  dtscc  ea  de  mi  grime  favorftC, 

Fite  jf  les  F^attES 


Marta  no  qu^rc  ccmprar  i2,  disco  perc  me  gusta  mucho  (2) 

Pablo: 

discc  que  ten  go,  que  ea  el  nuevc  CD  de  Paulina  Rubio 

Pdarta:  '  j0  PPdflmos  ccmprar  on  disco  de  (4)  [  arliata,  Odto  (E) 

mu  sica 

Pablo:  Es  evidenteoue  tenemes  que  encantraruna  sobciqn  L~t  gusta  (€; 
musca  qua  ernes  aftcra.  aquten  la  tiendfl" 

Marta;  ^  me  gusta.  Vamps  a  pregunlar  quien  es  y  vamos  a  ccmprar  G : .  _ 
disco. 


RMSH  Su*n  ■  forGracirg  vj 


Sub  mix  for  Grading 


CAPS  LOCK 
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APPENDIX  B 


Cadet  Feedback  Questionnaire 


1 .  Did  you  do  the  “Readiness  check"  for  this  Chapter?  _ yes  _ _ _ no 

2.  If  yes,  how  useful  was  it?  (please  circle) 

Very  useful  somewhat  useful  not  at  all 

3.  Please  circle  the  type  of  MySpanishLab  activities  you  completed  for  Chapter  5: 

L  Computer-graded  iL  Instructor-graded 


4.  Compared  to  the  hybrid  approach  of  previous  chapters  (Le.T  both  some  computer-graded 
activities  and  some  instructor-graded  activities),  do  you  believe  that  the  single  type  you  were 
assigned  this  chapter  enhanced  your  learning?  (Please  circle) 

Greatly  Somewhat  Neither  Enhanced  Somewhat  Greatly 

Enhanced  Enhance  nor  Detracted  Detracted  Detracted 


Comments: 

5,  Do  you  agree  with  the  following  statement;  "I  would  have  learned  as  much  if  I  had  the  other  type 
of  homework  activities.1’ 

Strongly  Agree  Neutral  Disagree  Strongly 

Agree  Disagree 


Comments: 


PARTICIPANTS  WITH  INSTRUCTOR-GRADED  ACTIVITIES  ONLY  (Cadets  with  Computer- 
graded  activities  please  go  to  Question  #9): 

6.  If  you  completed  the  Instructor-graded  activities  for  Chapter  5,  how  often  did  you  read  the 
instructor  feedback  provided  (please  circle): 


Every  Time  Most  of  the  Time  About  Half  the  Time  Rarely  Never 

7.  Do  you  agree  with  the  following  statement:  "The  instructor  feedback  helped  me  better  understand 
the  material  related  to  that  particular  activity.” 

Strongly  Agree  Neutral  Disagree  Strongly 

Agree  Disagree 


8.  Do  you  agree  with  the  following  statement:  “Overall,  the  instructor  feedback  enhanced  my 
learning  as  measured  by  my  performance  on  the  exam.1' 
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Strongly 

Agree 

Comments: 


Agree 


Neutral 


Disagree 


Strongly 

Disagree 


ALL  PARTICIPANTS: 

9.  What  do  you  like  most  about  MySpanishLab? 

10.  What  do  you  like  least  about  MySpanishLab? 

11 .  For  Chapter  5,  how  much  time  do  you  estimate  that  you  spent  on  MySpanishLab  activities: 
_ {m  in/hours) 

12.  Any  additional  comments  about  MySpanishLab: 

13.  If  given  the  option,  which  type  of  activity  would  you  prefer:  {please  circle) 

a.  Computer-graded  only 

b.  Instructor-graded  only 

c.  Hybrid  (both  types) 

Why?  Please  explain. 
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